1. The Field of the Invention
The present invention relates to the automated testing of computer systems. More specifically, the present invention relates to methods, systems, and computer program products for stochastically measuring the reliability of a computer system based on the expected usage by a variety of different users.
2. Background and Related Art
Computing systems are available in a wide-variety of forms including desktop computers and laptop computers. Even hand-held devices such as Personal Digital Assistants (PDAs) and cellular telephones have significant general-purpose processing and memory capabilities, especially when viewed from a historical perspective. Such hand-held devices may even run sophisticated operating systems that allow for the simultaneous usage of multiple software applications.
Current operating systems and supported applications can be quite complex. In order to operate properly, each application must interact harmoniously with the operating system (and with potentially other applications) so that the computer system functions as planned. It is often impractical to expect that a first pass at coding and compiling an application will result in a perfect finished product. Even a seemingly minor programming error may introduce significant performance deviations. With typical software applications often being compiled from many thousands of lines of code, one can appreciate that even the most experienced programmer will not catch every programming error in an application in the first instance. Accordingly, computer system and software providers extensively test their product to be sure the computer system works as intended.
One conventional approach to measuring software reliability is to send a partially complete software product out to a large population of users to test the product. However, it is often expensive to create a software product (even one with partial functionality) and distribute it to a large number of users. Additionally, sending such a partially complete software product may inadvertently allow competitors an advanced view of a product thereby reducing the product's competitive advantage.
Another conventional approach to measuring computer system reliability is to internally hire and train skilled testers to systematically exercise the system to find bugs. However, this approach to testing is highly labor intensive, and thus can be time consuming and expensive. Also, the skilled testers may not run the system through the same sequence of actions that a typical user might. Accordingly, some defects in the system may go unnoticed until the system is actually in the possession of the end-user.
In another conventional testing approach, computer-executable script is drafted that causes the system to perform a series of particular actions. While this approach does not necessarily require skilled testers to systematically exercise the system, the actions caused by executing the script will not necessarily approximate how an actual end-user might use the system.
Even if a software test accurately simulates how an actual user may use a system, there are many different types of users. For example, consider a system in the form of a PDA running phone, email, and Web navigation software. A regular consumer may use the phone software most frequently, with the email and Web navigation software being used less frequently. However, a traveling businessperson may use the email software most frequently, while using the phone a little less, and using the Web navigation software least of all. Accordingly, different user segments may use a system in dramatically different ways. Thus, even if a technique accurately tests a given user's use of the system, the technique may not be accurate for all users of the system.
Accordingly, what is desired are methods, systems, and computer program products for more realistically simulating system performance.